A mixed reality (MR) space presentation system presents an MR space image generated by compositing an image on a real space and that on an unreal space (virtual space) via a head-mounted display (HMD), thus allowing the user who wears the HMD to experience the MR space (or MR) (see H. Tamura, H. Yamamoto and A. Katayama: “Mixed reality: Future dreams seen at the border between real and virtual worlds,” Computer Graphics and Applications, vol. 21, no. 6, pp. 64-70, 2001).
In such MR space representation system, as a method of detecting the position and orientation of an image sensing device that captures the real space (or the position and orientation of a real object with respect to the image sensing device), a method using markers (also called landmarks) is known, as described in, e.g., Kato et. al.: Augmented Reality System and its Calibration based on Marker Tracking”, TVRSJ, Vol. 4, No. 4, pp. 607-616 (1999). In this method, objects (markers) each having a predetermined visual feature are laid out at known three-dimensional (3D) coordinate positions on the real space, markers included in a captured image are detected, and the position and orientation of an image sensing device are detected on the basis of two-dimensional (2D) image positions of elements (the barycenter, vertices, and the like of each marker) of the detected markers and the known 3D coordinate positions.
Shinji Uchiyama, Hiroyuki Yamamoto, and Hideyuki Tamura: “Robust Registration Method for Merging Real and Virtual Worlds -Combining 6 DOF Sensor and ICP Algorithm-”, Proc. I of Meeting on Image Recognition and Understanding (MIRU 2002), IPSJ Symposium Series, vol. 2002, no. 11, pp. I.107-I.112, 2002, discloses a method of repetitively correcting the position and orientation of an image sensing unit to minimize an error of corresponding points on a screen to have the position and orientation of 6 degree-of-freedom sensor, with a measurement error, as initial values.
Japanese Patent Laid-Open No. 11-136706 also discloses a method of obtaining the position and orientation of an image sensing device using markers and a position and orientation sensor in combination. In this method, a 6 or 3 DOF (degree-of-freedom) sensor, the measurement range of which is limited and which has measurement errors is fixed to an image sensing device, and position and orientation information measured by the 6 or 3 DOF sensor and position and orientation information obtained by marker detection on a captured image are combined, thereby improving the measurement precision.
In any of these methods, the precision of the obtained position and orientation largely depends on the marker detection precision. For example, if a plurality of regions recognized as markers from a captured image include at least one erroneously detected one, the finally obtained precision of the position and orientation impairs considerably. As a factor that induces such detection errors, an object having an image feature similar to each marker may exist on the captured real space.
In order to avoid erroneous detection of a region having a similar feature other than markers, a method of using markers having visual features (shapes, patterns, colors, and the like) having a lower probability of existence on the real space is often adopted.
However, as factors that lower the calculation (estimation) precision of the position and orientation using markers, those other than “erroneous detection of an object other than markers” are present: for example, the precision drop resulting from a marker partially occluded by another object so as not to disturb detection (to be referred to as a partially-occluded marker hereinafter). Since the partially-occluded marker is detected as a region having a shape different from a region to be detected, the barycentric position and the coordinates of vertices are erroneously recognized or calculated, thus deteriorating the final position and orientation calculation precision of an image sensing device. In the following description, a state wherein marker information cannot be correctly obtained due to partial occlusion of a marker will be referred to as erroneous detection due to partial occlusion.
FIGS. 5A to 5C and FIGS. 6A and 6B are for explaining examples of erroneous detection due to partial occlusion. In FIGS. 5A to 5C, a marker (rectangular marker 4000A) includes a rectangle 5020. When such marker is used, four vertices (5030, 5040, 5050, and 5060) of the rectangle 5020 must be correctly detected.
However, when the two vertices 5030 and 5040 of the four vertices of the rectangle 5020 are occluded by a user's hand 4020R, and the rectangle 5020 is detected by an image process (e.g., binarization) shown in FIG. 5B, contacts 6010 and 6020 between the hand 4020R and rectangle 5020 are erroneously recognized as the vertices 5030 and 5040 (FIG. 5C).
FIGS. 6A and 6B show a case wherein a marker (color region marker 4010) shown in FIG. 6A is partially occluded by the user's hand 4020R, as shown in FIG. 6B. In this case, a wrong position 7010 is detected in place of an original position 7000 as the barycentric point of the marker.
The position and orientation of the image sensing device using markers are calculated based on correspondence between the known vertices or barycentric position of a marker (or a pattern of the marker) on the world coordinate system, and coordinates on the coordinate system of the image sensing device. Therefore, when the feature points (vertices, barycenter, or the like) of the marker detected in an image are erroneously detected, the precision of the finally obtained position and orientation information impairs.
However, with the conventional marker detection method, erroneous detection due to partial occlusion cannot be determined even if it has occurred.